Skip to content

feat(api): add the indexer and flusher bounds the chain cutover needs - #1028

Open
rickyrombo wants to merge 1 commit into
mainfrom
mjp-cutover-plumbing
Open

feat(api): add the indexer and flusher bounds the chain cutover needs#1028
rickyrombo wants to merge 1 commit into
mainfrom
mjp-cutover-plumbing

Conversation

@rickyrombo

Copy link
Copy Markdown
Contributor

What

Three config values, all defaulting to 0 meaning unbounded, so nothing changes in normal operation. They exist for the genesis-migration cutover (cmd/genesis-writer/ROLLOUT.md, Runbook step 12), which cannot be executed without them.

env effect
etlStartingBlockHeight passed to the ETL's SetStartingBlockHeight
etlEndingBlockHeight passed to the ETL's SetEndingBlockHeight
newChainFlushToBlock ceiling on the flusher — the mirror of the existing newChainFlushFromBlock

Why the indexer bounds

indexer/indexer.go calls SetConfig, SetDBURL, SetCheckReadiness and the hooks — but never the two height setters, even though the ETL has supported them all along. So the indexer only knows how to resume, and the resume query is:

SELECT MAX(block_height)::bigint FROM etl_blocks HAVING MAX(block_height) IS NOT NULL

That has no chain_id. Pointed at a new chain it resolves to the old chain's height — roughly 24M against a chain at a few thousand — and polls for a block that won't exist for years. It fails as a silent stall, not an error.

There is a chain-aware fallback to core_indexed_blocks WHERE chain_id = $1, but it only runs when GetLatestIndexedBlock returns ErrNoRows. On any database that has indexed the old chain, etl_blocks is populated, the first query answers, and the fallback never fires. That's precisely why the failure is quiet — the code that would have caught it is one branch away.

The ending bound is the other half: the cutover needs the old-chain indexer to stop at a known height L, chosen in the future so the stop doesn't race the config rollout.

Why the flusher ceiling

The cutover needs everything confirmed at or below L to be on the new chain — and nothing above L across that line — before the new indexer starts at H+1. Without a ceiling the flusher would carry rows above L over the boundary, and those would be indexed from the old chain and from the new one.

It filters rather than halts, and that distinction matters. Enqueue is dispatched with go app.enqueueForNewChain(...), so confirmed_block is only roughly ordered by id — two writes confirmed at blocks 100 and 101 can be inserted in either order. Halting at the first row above the ceiling would strand a row below it, which would then flush after the boundary was recorded and be indexed twice.

NULL confirmed_block is held while a ceiling is set. Such a row can't be placed relative to L; holding is the recoverable choice, since it flushes once the ceiling lifts, whereas sending it early could double-index and dropping it would lose the write. Worth draining or inspecting those before a cutover rather than discovering them during one.

Tests

Flusher, against a live test database:

  • boundary is inclusive — rows at exactly the ceiling are eligible
  • filters past an out-of-order row — a low-id row above the ceiling doesn't hide a high-id row below it
  • NULL confirmed_block is held, then released when the ceiling lifts
  • no ceiling configured ⇒ unbounded, as today

Confirmed failing without the change:

--- FAIL: TestNewChainFlusherCeilingHoldsRowsAboveIt
    should have 2 item(s), but has 4
--- FAIL: TestNewChainFlusherCeilingFiltersRatherThanHalts
    expected: []int64{2}   the row below the ceiling must be reachable past one above it
--- FAIL: TestNewChainFlusherCeilingHoldsNullConfirmedBlock
    Should be empty

Plus a config test for the shared parse helper. The three bounds all mean "no bound" at 0, so a malformed value must panic rather than read as 0 — otherwise a typo silently disables the very limit it was meant to impose, and that only surfaces as duplicated or missing rows much later.

Pre-existing flusher tests still pass.

Note

Touches api/new_chain_flusher.go, which #1018 also changes. Whichever lands second will need a small rebase — they don't overlap logically (that one changes how rows are retired, this one changes which rows are eligible).

Three config values, all defaulting to 0 meaning unbounded, so nothing changes
in normal operation.

etlStartingBlockHeight / etlEndingBlockHeight are passed to the ETL's existing
SetStartingBlockHeight / SetEndingBlockHeight, which indexer.go never called.
Without them the indexer only knows how to resume, and the resume query --
MAX(block_height) FROM etl_blocks -- carries no chain_id. Pointed at a new chain
it resolves to the old chain's height and waits for a block that will not exist
for years, silently. There is a chain-aware fallback to core_indexed_blocks, but
it only runs when etl_blocks is empty, so it never fires on a database that has
indexed the old chain.

newChainFlushToBlock is a ceiling on the flusher, the mirror of the existing
newChainFlushFromBlock. The cutover stops the old-chain indexer at a height L and
needs everything confirmed at or below L to be on the new chain -- and nothing
above L across that line -- before the new indexer starts.

The ceiling filters rather than halts. Enqueue is dispatched asynchronously, so
confirmed_block is only roughly ordered by id; stopping at the first row above
the ceiling would strand one below it, which would then flush after the boundary
was recorded and be indexed twice. Rows with a NULL confirmed_block cannot be
placed relative to L and are held until the ceiling lifts.

Tests cover the ceiling boundary (inclusive), that it filters past an
out-of-order row, that NULLs are held and then released, and that an unset
ceiling is unbounded -- confirmed failing without the change. A config test
covers the parse helper, since a malformed bound reading as 0 would silently
disable the limit it was meant to impose.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant